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Do government policies work as they are supposed to? This is not just a hypothetical question, but a 
question we can actually answer by collecting careful data. Some few states around the U.S. are beginning 
to do just that— test whether their policies are actually working— and the current report is about one of 
Wisconsin’s efforts in this direction. 


The current report tells us about the effectiveness of one program of Wisconsin state government, the 
YoungStar system which provides consumers with a simple 5 Star rating system for the quality of early care 
and education (ECE) programs for children ages 0 to 5. The study goes beyond asking if the program was 
delivered as promised, and asks a much more difficult question: were the lives of citizens changed in the 
ways we hoped because of this program? 


The study was well done, rigorous. The evaluation was led by Dr. Katherine Magnuson of UW-Madison, 
and she is one of our nation’s leaders in research on ECE programs (we are fortunate to have her here in 
our state). This wasn’t a study of opinions. Magnuson’s team conducted observations in ECE programs 
across the state to rate the quality of their teaching interactions with young children, and they also directly 
tested the child development gains of young children in such areas as vocabulary growth, impulse control, 
behavior problems, and their general readiness for school. 


The summary that follows, prepared by David Edie at the Wisconsin Council on Children and Families, 
accurately pinpoints the key findings and their implications. The most important finding was that the state’s 
YoungStar system really does reflect differences in quality: programs with more stars were doing a much 
better job with young children. To give an idea of what this means in practice, take the area of language 
and reasoning. In the average 5 Star program you would observe teachers introducing logical concepts (for 
example over/under, before/after) into conversations with children, and you would also see that children are 
encouraged to talk through or explain their own reasoning when solving problems. Some 2 Star programs 
also did these practices with children, but the average 2 Star program did not. These are the kinds of 
practices, of course, that help children developing their thinking skills so they will later succeed in school 
and in life. In other words, these differences make a big difference in the children’s lives. 


But another of the findings seems disappointing. Children in higher quality programs did not gain 
substantially more from fall to spring than did children in lower rated programs. This certainly flies in 

the face of hundreds of studies, which show really dramatic impacts of high quality ECE upon children’s 
development. These impacts are so well researched, and so dramatic, that even economists from our nation’s 
Federal Reserve Bank are saying we should invest more in high quality ECE programs because they 

save the government more than they cost in the long run (especially in reduced costs of crime and public 
supports). 


But in two ways the current findings are really not that surprising. First, this evaluation observed the 
children in the middle of their child care careers. Presumably, those who had been in higher quality 
programs for a year or two before this study would have gained more in their child development prior to 
the study, and the data are consistent with that idea. In the fall, when the children were first tested, those 
in higher quality programs were doing significantly better in their language and math skills, and then the 
children in all programs gained in ability at similar rates across the year. 


Second, many other studies have found that ECE enrichment programs have effects that appear very 
small at first, but if you continue to follow the children the impacts become quite dramatic by the late 
teenage and early adult years. A small deflection in one’s life course at this early age, especially in the 
areas of intellectual development and self-control, can add up to quite different life outcomes over the 
course of many years. So the fact that lower and higher quality programs produce similar changes in child 
development across the year is not really that much of a surprise. We need to follow the children for more 
years to really see the impacts of different programs. 


One impact that was not studied in this evaluation, but will interest policy makers, is the impact of 
YoungStar on the marketplace for early care and education. The YoungStar program was designed in part to 
accentuate the naturally occurring forces of the free market economy. Prior research showed us that parents 
want to trust their children only to high quality settings, but parents don’t know how to gauge the quality of 
programs. The YoungStar program, by providing an easy measuring stick of quality, was designed to help 
parents and thus to create consumer pressure upon programs, to raise their number of stars so that more 
parents would want to purchase their services. In turn, waiting lists at the highest quality programs should 
allow them to raise their fees and pay their teachers better, which should reduce the problem of worker 
turnover and keep well trained teachers in the profession longer. In this way, YoungStar should help the 
marketplace work more efficiently to raise the quality of early care and education programs across the state. 
Indeed, other studies have already shown this to be happening. 


YoungStar works. But could we improve it? Yes, we certainly could, but it is not clear that we should. The 
current YoungStar system was designed to be easy and inexpensive for government to administer, and in 
this it has succeeded. For example, it does not include the rigorous but expensive direct observations that 
Dr. Magnuson used in her evaluation study. Inevitably, we will need to choose a Quality Rating System 
that compromises between accuracy and cost, and the current version is not bad on that score. Nonetheless, 
the state should periodically consider fine-tuning the YoungStar system of awarding stars for quality. 

For example, if the state wished to emphasize pre-reading skills, or math foundations, or the prevention 

of problem behaviors in young children, then the YoungStar system could be adapted to each priority. 
Behavioral scientists already have the expertise to do this— it is just a matter of policy makers setting the 
priorities for the development of our next generation of Wisconsin citizens. 


David Riley is a retired professor in Human Development and Family Studies at the University of 
Wisconsin-Madison and UW-Extension. Dr. Riley was recently honored for “exceptional contributions to the 
fields of early childhood and parenting, particularly by producing a measurable, positive impact on issues of 
concern to Wisconsin citizens,” by the Board of Human Sciences. A major part of his work focused on high 

quality child care and effective parenting. 
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This report will explore what was learned from a 

recent research study of YoungStar, Wisconsin’s child 

care quality rating and improvement system. The re- 

port will address key questions, including: 

¢ Is the 5 Star quality rating system valid? 

¢ Do program ratings improve over time? 

¢ Do programs with higher quality ratings produce 
better outcomes? 

* How well has Wisconsin done in its effort to build 
a strong Quality Rating and Improvement System 
(QRIS)? 


Youngstar Purpose, Structure, and Scope 


In the last six years Wisconsin has been building a 


new system to increase the quality of child care, with a 


particular focus on children from low-income working 
families. The new system is called YoungStar, a Qual- 


ity Rating and Improvement System (QRIS) and rolled 


out in the beginning of 2010. YoungStar is a program 

of the Wisconsin Department of Children and Families 

(DCF) created to improve the quality of child care for 

Wisconsin children, with these key objectives: 

¢ Evaluate and rate the quality of care given by child 
care providers; 

¢ Help parents choose the best child care for their 
kids; 

¢ Support providers with tools and training to de- 
liver high-quality early care; and 

¢ Seta consistent standard for child care quality. 


Decades of research have emphasized the importance 
of early childhood development. The hope was that 
YoungStar would harness marketplace forces to in- 
crease the quality of child care programs statewide. 
YoungStar evaluates the quality of care given by 
regulated child care providers and rates them at | to 5 
Stars, with 5 Stars being the highest rating, similar to 
ratings for hotels, restaurants, or other types of busi- 
nesses. 


A provider’s star rating is based on these four do- 
mains: 

1. Educational Qualifications and Training 

2. Learning Environment and Curriculum 

3. Professional and Business Practices 

4. Child Health and Well-Being Practices 
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In each domain programs can earn “points” to reach 
higher ratings. 


Department of Children and Families 
YoungStar Rating Guide 


Meets highest levels of quality standards 


Meets elevated levels of quality standards 
| 3Star (Meets proficient levels of quality standards 
Meets health and safety standards 


Does not meet health and safety standards 
1 Star (and therefore cannot participate in the 
Wisconsin Shares reimbursement system) 
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YoungStar is one of the most developed quality rating 
systems in the country, with 3,800 programs par- 
ticipating, and serving 43,000 low-income children. 
Many states’ QRIS programs reach only a fraction of 
their child care providers. In Wisconsin, over 70 per- 
cent of licensed child care programs are participating 
in YoungStar, and all children subsidized by Wiscon- 
sin Shares are in YoungStar. Since most programs 
serving subsidized low-income children also serve 
non-subsidized children, it is likely that YoungStar has 
an impact far more than 43,000 low-income subsi- 
dized children, possibly as many as 100,000 children. 
Clearly, YoungStar has the breadth and scope to have a 
significant influence on young children and their fami- 
lies. Detailed information on YoungStar can be found 
on the DCF website: http://dcf.wi.gov/youngstar/. 


What Have We Learned from Recent Research 
on Quality Rating and Improvement Systems? 


Before delving into Wisconsin’s study, it is useful to 
understand related research to date. The most thorough 
overview of QRIS research to date was completed 

in 2014 by RAND Education and RAND Labor and 
Population, Validation Studies for Early Learning and 
Care Quality Rating and Improvement Systems. The 
RAND study reviewed 14 QRIS rating systems in 

11 states: Colorado, Florida (two counties), Indiana, 
Maine, Minnesota, Missouri, North Carolina, Okla- 
homa, Pennsylvania, Tennessee, and Virginia. The 
RAND study focused on three validity questions: 


1. Do programs with higher ratings have higher 
observed quality? 

Of 11 studies, 10 studies provided evidence that 
the ratings in the QRIS programs are “capturing 
meaningful difference in program quality.” In sev- 
eral QRIS programs, highly rated programs were 
measured higher by Environment Rating Scales 
(ERS), but ERS measures were embedded in the 
rating levels. 

2. Do programs’ ratings improve over time, as 
would be expected with the help of technical as- 
sistance and other supports? 

In six studies focusing on this topic, there was a 
consistent finding that quality increased over time 
among participating providers. 

3. Do programs that receive higher ratings pro- 
duce better outcomes in child development? 
The RAND study found that “these studies are the 
most challenging to implement and can be costly 
to conduct when independent child assessments 
are performed.” Among seven studies, RAND 
finds “only limited evidence that programs rated 
more highly in a given QRIS are associated with 
larger development gains for the enrolled chil- 
dren.” Overall, there were mixed results about 
whether higher ratings predicted better child devel- 
opment outcomes. Two programs found significant 
effects in social and behavioral development, and 
in pre-literacy assessment (assessed by teachers). 


The background from the RAND study gives us a 
frame as we look at Wisconsin’s study. QRIS is a 
fairly new public policy across the country, and the re- 
search efforts are in the early stages. In short, research 
to date shows that programs with higher QRIS quality 
ratings have higher observed quality, that QRIS ratings 
improve over time, but that there were mixed findings 
of the impact of a higher rating on producing better 
child development outcomes. 


The RAND QRIS Study focused on three 
questions: 


1. Do programs with higher ratings have higher 
observed quality? 


2. Do programs’ ratings improve over time, as 
would be expected with the help of technical 
assistance and other supports? 


3. Do programs that receive higher ratings produce 
better outcomes in child development? 


Wisconsin's QRIS Validation Study 


At the end of 2012 Wisconsin was one of 14 states 
awarded a federal Race to the Top-Early Learning 
Challenge (RTTT-ELC) grant, competing among doz- 
ens of states. The grant required that some of the funds 
should be used for building the state’s QRIS, which 
helped Wisconsin improve the rating system, increase 
technical assistance, and help programs increase their 
quality. Along with the federal funds came a require- 
ment for each funded state to complete a QRIS valida- 
tion study. Two federal agencies, the Department of 
Health and Human Services and the Department of 
Education, wanted to learn more about the progress 

of QRIS in the funded states. The grant required that 
each state complete an evaluation, working with an 
independent evaluator, with two major questions: 


1. Do the tiers in the state’s tiered quality rating and 
improvement system accurately reflect differential 
levels of program quality? 


2. To what extent are higher quality ratings related to 
progress in children’s learning, development, and 
school readiness? 


The research reports on the validity of YoungStar’s 
rating scale were finalized in May 2016. The research- 
ers were Katherine Magnuson, PhD, a prominent 
scholar in early child development nationwide, and 
Ying-Chum Lin, MSW, at University of Wisconsin- 
Madison, School of Social Work and the Institute for 
Research on Poverty. The research included 887 chil- 
dren in 157 early care and education programs. The re- 
search began in the fall of 2013 and went through the 
spring of 2014. The child care programs participating 
included 204 group child care centers and 34 family 
child care providers. Most programs and classrooms 
in this study were in the 2 Star and 3 Star categories. 
Only 12% of the classrooms were in the higher-rated 
programs (4 Star and 5 Star). 


Because YoungStar was still in its early stages at the 
time data collection began, only 1% of programs in 
the state were at 4 Star and only 7% were at 5 Star, 

so there wasn’t a broad range of the highest-rated 
programs to participate in the study. The children 
involved were ages 3 to 5. The child care programs 
selected were in the Northeast and Milwaukee regions 
of the state. 


Breakdown of Early Education Classrooms 
Observed in YoungStar Validation Study 


22 


| eis | 108 | 102 | 7 | 22 


% of Total 
Classrooms 
Observed 
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While YoungStar was designed particularly to focus 

on low-income children subsidized by Wisconsin 
Shares, only about one third of the children tracked in 
the study were from low-income families, mostly par- 
ticipating in Wisconsin Shares. Shares data shows that 
92% of participating families are single-parent fami- 
lies in poverty or near poverty, quite different from the 
rest of the sample. 


The validation study released three separate reports: 
The Executive Summary of the Wisconsin Early Child 


Care Study 


Wisconsin Early Child Care Study First Component 
Full Report 


Wisconsin Early Child Care Study Second Component 
Full Report 


What Did We Learn from the Wisconsin 
Validation Study? 


1. The YoungStar ratings differentiate programs 
based on observed quality. 
The YoungStar star rating level does differenti- 
ate among programs of varying observed quality. 
The results found substantial differences between 
2 Star and higher-rated programs (3 Star, 4 Star, 
and 5 Star). According to the researchers, the 
“findings provide the first investigation of empiri- 
cal evidence that observed quality is found to be 
higher among 3 Star or higher rated programs than 
for 2 Star programs. The research also found that 
the quality ratings of most child care providers are 
in the minimal to good range.” The study did not 
have a large enough sample of 4 Star and 5 Star 
programs to differentiate among the full range of 
programs (2 Star through 5 Star). 
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2. The YoungStar rating “points” system predicts 
a program’s observed classroom quality. 
The study also found that the quality points system 
though which programs earn “points” appears to 
be valid. The total number of rating points in each 
of the four domains (educational qualifications, 
learning environment/curriculum, professional 
and business practices, and child health) predicted 
program quality. 


3. On average, children in YoungStar programs 
were meeting national norms. 
The study found that there was a clear pattern: 
overall, children in all types of programs were at 
or above the national norms of school readiness. 


4. Children are making progress. 
The study concluded that on average children 
in YoungStar programs were learning and mak- 
ing age appropriate gains, and learning a range 
of skills during the course of the preschool year. 
It’s good news that overall children were making 
progress. The research suggests that on average 
these children were likely to enter formal school- 
ing ready to learn. 


5. Higher YoungStar quality ratings (on the 5 Star 
scale) did not predict higher levels of school 
readiness over the period of the study. 

The study showed that the lack of association 
between the YoungStar rating and children’s 
outcomes was found across academic skills. The 
researchers suggested that the “broader dimen- 
sions” of child care quality assessed by YoungStar 
were not the key inputs that increase children’s 
pre-academic skills and learning behaviors. The 
researchers found there was a “lack of association 
between YoungStar rating and child outcomes.” 
The researchers indicated that these findings are 
“consistent with most validation studies of other 
state or local QRIS ratings systems that have ex- 
amined children’s outcomes.” 


The research findings are confusing: how can higher 
quality not have better outcomes? The researchers 
suggested that “higher quality child care, within the 
range of moderate to good care, is necessary but not 


sufficient for intentionally and specifically developing 
children’s early school readiness.” It is understandable 
that higher rated programs did not have significant 
child outcomes in the short timeframe, but why did 
children in all rated programs, even with the lowest 
ratings, meet national norms on average? 


As the researchers indicated, this finding is not sur- 
prising based on research in other states. QRIS is 
fairly new for most states, and research on outcomes is 
in its early phases. So far most studies of QRIS have 
not found strong improvement in children’s outcomes 
based on the ratings system alone. There are several 
possible reasons that contributed to the finding that the 
quality ratings didn’t predict child outcomes: 


¢ The small sample in programs rated 4 Star and 5 
Star. Only 12% of the classrooms sampled were 
in programs rated at the highest levels. A larger 
sample of 4 Star and 5 Star programs could have 
resulted in different findings. 


¢ The sample included only about one third that 
were low-income children or disadvantaged. 
YoungStar particularly focused on improving 
child development for children in low-income 
families. That’s why child care programs serving 
subsidized-children are required to participate in 
YoungStar. It’s possible that a larger sample of 
low-income children in 2 Star and 5 Star programs 
might have led to different results. 


¢ The timeframe is less than a year. The timeline 
may not have been sufficient to show significant 
changes, especially when many of the children 
had already had been exposed to quality programs 
prior to the research project. The largest impacts 
of good early care and education (ECE) programs 
take time to accumulate. Even the studies finding 
the most dramatic impact of ECE programs have 
tended to find minimal differences after one year. 
But small differences early in life can add up to 
very large eventual difference in schooling and 
lite, 


* Quality ratings may be necessary but not suf- 
ficient. It is hard enough to find solid child out- 
comes with a consistent early learning program 


with highly qualified teachers. But when there are 
various child care programs using a wide range 
of approaches and curricula, it is difficult to get 
consistent results in child outcomes. QRIS pro- 
vides a broad framework for quality, but it may be 
that “broad dimensions of child care quality as- 
sessed by YoungStar are not the key inputs,” as the 
researchers suggest. As the researchers indicate, 
solid quality ratings are necessary (as in Young- 
Star), but not sufficient, to get the child outcomes 
desired. For instance, some researchers believe 
that the key to early child development is warm 
interactions and rich conversation between adults 
and young children, along with good instructional 
practice. There are now new ways to measure 
those critical interactions that set the foundation 
for children to thrive (for instance, CLASS mea- 
sures the effectiveness of teacher-child interac- 
tion). But of course, more assessments can be 
costly to implement. 


A Balancing Act for the State: High Validity Versus 
Low Cost 


The current QRIS developed by Wisconsin strikes an 
efficient balance between low cost and high validity. If 
we change YoungStar so that it requires direct obser- 
vations of the interactions between teachers and chil- 
dren, which is the key to child development, we could 
make much stronger predictions of child development 
gains. But this would be at a great cost to government. 
The current QRIS, while imperfect, might strike the 
best balance between efficiency and validity. 


Difficulty in Attracting and Retaining Excellent 
Teachers and Administrators 


Child care programs nationwide continue to struggle 
to attract and retain excellent teachers. While Young- 
Star resources and scholarships help, well-trained Wis- 
consin child care teachers (with an associate degree or 
a bachelor’s degree) on average make only $10 to $12 
per hour, according to a 2015 Wisconsin’s Child Care 
Workforce study by the Wisconsin Early Childhood 
Association. One in three employees in child care 
centers leave their jobs each year, so any effects of 
staff training are reduced through attrition each year. A 
stable, well compensated workforce is key to quality 
in child care. Most researchers would agree that effec- 
tive, experienced teachers and their interaction with 
children and their families are essential for positive 
child outcomes. 


What Are the Implications for YoungStar? 


Wisconsin Should be Proud of Building a Strong, 
Robust, Valid YoungStar Foundation 

After the careful planning to create a rating system 
that was evidence-based, it is encouraging that Young- 
Star differentiates observed quality. Since YoungStar is 
using the 5 Star system to inform parents, and the state 
provides different levels of payments based on the rat- 
ings, the study validates YoungStar’s structure. It also 
confirms the effort many public and private individu- 
als put into launching YoungStar. The rating system, 
with its points system, has been validated by the study. 
While we don’t know exactly what is helping children 
to be at and above national norms for school readiness, 
it is comforting to know that the children in YoungStar 
have a solid start. The Department of Children and 


November 2016 


Families, the private agencies that implemented much 
of the QRIS, and the early care and education pro- 
grams across the state have helped Wisconsin meet its 
original four goals: 


— 


YoungStar has evaluated and rated the quality of 
3,800 child care providers, a tremendous achieve- 
ment. 


2. YoungStar has created a system that helps thou- 
sands of parents choose the best child care for their 
children, with access to a website with quality 
information on all YoungStar programs. 


3. YoungStar has supported providers with extensive 
technical assistance, quality achievement grants, 
and scholarships for thousands of teachers to in- 
crease their proficiency. 


4. YoungStar has set a consistent standard for child 
care quality statewide, providing quality bench- 
marks that programs and communities can strive to 
meet. 


YoungStar has created a strong foundation to build on. 
It has made excellent programs, but it is not likely that 
the child care programs participating in YoungStar can 
be transformed into model early learning programs 
without ongoing resources to sustain well-trained, 
well-paid teachers. 


Next Challenge for YoungStar: Focus on Child 

Outcomes 

Wisconsin and states across the country with a QRIS 

are focusing on child outcomes and school readiness. 

YoungStar has set up a solid, valid system to build on. 

Wisconsin needs a mechanism to measure progress in 

school readiness. For instance: 

* KEA: Unlike many states, Wisconsin has not 
developed a statewide Kindergarten Entrance As- 
sessment (KEA) to have a broad view of children’s 
school readiness. Solid data might help the State 
focus resources where they are most needed. Wis- 
consin has looked at several options for KEA, but 
current policy leaves this up to each school district. 


¢ Data System: Wisconsin has been building an early appear to be already meeting national school norms on 
childhood longitudinal data system that could track average. Wisconsin should particularly focus on po- 
children from early childhood into public schools __ tentially vulnerable children from low-income families 
to help indicate what programs are most effective. and children of color, and their access to 4 Star and 5 
Star child care programs, and ensure opportunity for 


The challenge for Wisconsin and most other states those that need it most. To make progress, Wisconsin 
is to improve the quality of child care sufficiently can learn from many other states and from additional 
to result in positive child outcomes, particularly for research, as YoungStar moves ahead toward 2020. 


children from low-income backgrounds or with high 
needs. To reach that goal, more public and private 
investment will undoubtedly be needed. The good 
news is that children in YoungStar child care programs 


WISCONSIN COUNCIL ON 


children 


& families 


Raising Voices to Make Every Kid Count 


www.wccf.org 
www.facebook.com/wiskids 
www. twitter.com/wiskids 


